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AB'STIACT 

Because reduction of energy has become a major requirement 
foT water and wastewater treatment systems design, introduction O'f an 
energy criterion has increased the number of design variables and 

decisions to be made for plant design and evaluation. To optlinize a 

plant consisting of many stages and recycles, a means of evaluating 
performance with respect to effluent quality, operating co'St and 
energy must be utilized. This report suggests that the technique of 
"dynamic programming" be applied as a means of system optimization. 
Several examples are used to illustrate the application of this 
optimization method. 
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1, Ini t r o due t ion 

The problems associated with energy and energy resource systems 
which have risen in the past two years have led to intensive efforts to 
reduce energy comsumption. Water and wastewater treatment plants are now . 
being scrutinized for means of reducing, energy demands and losses. 
Earlier investigations by the Applied Sciences Section of the Pollution 
Control Branch produced a number of reports on energy requirements for 
water and wastewater systems involving overall energy requirements for 
treatment systems as well as the requirements for individual unit 
operatlo'ns and processes (Zarnett, 1975 , r9'76„ 19 7 6a) , The data from 
these studies can be utilized for design purposes in evaluating alternatives 
for new plants in which energy expenditure is to be minimized. 

In previous years, the general design problem involved minimizing 
plant capital cost and operational cost and, at the same time, achieving 
the required effluent standards for which the system was designed. Now 
that energy reduction has become a significant goal, a means of plant 
evaluation is necessary in which cost, effluent requirements and energy 
requirements are optimized. This added dimension in design and planning 
requires a multitude of new decisions to be made along with a means to 
evaluate them,. The purpose of this proposal is to suggest a means o^f 
systematically making decisions which will lead to an optimal energy 
policy. 
2. Multistage Decision Procedure 

Any planning problem, be it water treatment plant design, 
manufacturing plant layout, corporate expansion program, or family vacation 
trip. Involves many decisions, each affecting the other. In planning 
problems, the components may be arranged in series, each representing 
the same technological complex at a different point in time,. This serial 
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structure,. In which decisions at any stage affect the conditio'ns of all 
later ones, o-ccurs alsO' in prO'Cesses having manufacturing units in series. 
Several systems lend themselves to^ decomposition by what is termed partial 
optimization. The earliest and best-kno^wn example of the partial 
optimization approach is Bellman's, "dynamic programming" (B^ellman, 1957) » 
■"Dynamic" Is used in the sense of being novel, and "prograinming" is used 
to describe the technique as being procedural or algorithmic, and Is not 
related to computer prograiraiing, although the method, due to its algo- 
rithmic nature, can utilize a computer very effectively. 

This method has. the capacity to resolve problems in optimization 
which have ""real world" constraints applied to them which were formerly 
ignored or unsolved, and to attack problems involving multidimensional 
dec le Ions. 
3. Dynamic Programming 

Dimamic programming embraces the area of multistage decision 
processes. A, multistage decision process Is one, as the name Implies, 
with many stages Involving decisions.. In particular, in dynamic prograimiing, 
the decisions must be made in the light of the entire process rather than 
in the light of each s.tage as an entity. This me.ans the decision at each 
stage must be "right" in view of the entire process. 

The main concept of this technique lies in the "prliniciple of 
optlmality" (Bellman, 1957) : 

An optimal policy has the property that, whatever the initial 
state .and the Initial decision, the remaining decisions must 
constitute an optimal policy with regard to the state resulting 
from the first decision. 
This principle of optlmality is an intuitive concept and c.an be used as 
such with no mathematics beyo^nd arithmetic or can be involved in solving 
complex mathematical models. Early research has shoOT that this Intuitive 
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form can be reduced rigor O'usly to an exact mathematical form such as 

the Euler-Legran,ge equatlo'n of calculus of variations, or to Pontryagin's 

maximum principle (Bellinan, 1962; Lapidus & LuuSj 1967). 

If O'ptliiilzatlon is to be per formed,, it should be regarded as a 

multistage decision process in which at each stage the control variable 

V is to be chosen in terms of the current state denoted by x , and the 
n n 

number of stages remaining in the process. This involves abstracting the 

mathematical content of the fundamental engineering principle of feedback 

control . The influence exerted upon the system, is dependent upon the 

current stage of the systeitt. This is an example of event orientation. 

If we start in state x = c and choose the control vector y„, as In 

o „ o 

1, ^. '\' 

FiRure 1, and define a criterion function J = h(x , y ), the result of 

% 1. 

decision y is h(x , y ) . 

■^ o ^ o o 



current state 

X 







Figure I 



Starting with the new state x , we choose y ; see Figure 2. The new 

or ^ 

contribution to J is h(x ^ y ). 

current state 

X, 
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Figure 2 
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This maj be considered to be a multistage decision process with 
an ii^ortant invariant structure. Due to the additivity of the criterion 

function at each stage, the past (both the past history of the process 
and the past costs) may be ignored when it comes to making present and 
future decisions. We are free to concentrate on choosing the remaining 
variables, so as to minliiize the remaining part of the criterion function, 
by starting with a knowledge of the current state. 

It is this sharp decomposition into a past, present and future 
which permits obtaining an analytical hold on the optimization process. 

Policies 

A declslo^n made in state C, when there are N stages remaining,, is 

written as y(C,;N) . In general, a function y(C,N) which yields an 

admissible choice of y Is called a policy. A policy which yields the 
— ' o 

absolute minimum (or maximum) of the criterion function is called the 

optimal policy . Policy is a basic co^ncept of dynamic programming. 

The Performance Ittdex 

To measure the "goodness-of-control" in some meaningful manner, 
it is necessary to Introduce a criterion which specifies how well the 
system is doing. In alternative terms, we wish to specify how close the 
control achieves any goal which se set up for it. This is accomplished 

by means of a scalar performance index . A typical example of a performance 
index is the air pollution, index. If the pollution index is exceeded some 
control Is usually applied to reduce or minimize the performance index. 

For water and wastewater treatment design, the performance index 

should include operating costs, degree of contaminant removal and energy 
requirements. Each of these parameters should be weighted against each 
other for a realistic design criterion- This type of performance Index 



- 5 = 



can be easily incoTpoTated into the dynaniic programming fo'rmulatio'n. 

A moTe complete discussion may be found elsewhere., (Athans & Falb, 1966). 

4. A Multistage Qptimiza-t_iOTi_Sxs_tem 

A simple example Is shown schematically in Figure 3. A waste- 
water treatment system is to be designed on the basis of a minimum of 
energy O'nly, with nO' consideration of advantages or disadvantages O'f the 
unit operations and pro'Cesses involved or any government regulations or 
guidelines. The purpose of this simplified example is to illustrate the 
principle of optimality. The numbers stown with each operation is the 
performance index which in this case is only the unit energy requirenient 
In kwh/d. Taking the final effluent as the origin = 0', the analysis 
proceeds stagewise backwards suinmlng the performance indices at each 
stage. At the originj I - 0'; at stage (1) from the right, I - 303, 935, 
462. 

E min 1(0,1) = mln (303, 935, 462) +1(0,0) = 303 + - 303 
For stage (2), E mln 1(0,2) - min {13410, 333) + E 1(0,1) = 333 + 303 - 636 

(3), 1 min 1(0,3) - min (13410, 6337, 4368) + E 1(0,2) 

= 4368 + 636 = 5004 

(4), Z min 1(0,4) = mln (13410, 1150, 5743, 3666) + E 1(0,3) 

- 1150 + 5004 = 6154 
(5), mln E 1(0,5) = min (370, 50, lOO'O, 20000, 375, 75)+ E 1(0,4) 

- 50 + 6154 = 6204 
(6), min E 1(0,6) = 4.93 + E 1(0,5) = 620i9 

Therefore, the iilnlni,i.mi energy requirement is 6209 kwh/d, and the plant 
should consist of screening, sedimentation, stabiliiation basin, aimiionia 
stripping, chemical precipitation and ozonation. 

This is a simplified example which can be done mentally, but 
illustrates the technique of dynamic programming. A more cooipllcated 
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FIGURE 3 10 mgd WASTEWATER TREATMENT SYSTQl 
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process is slmwn. In Figure 4 in which pumping and auxilliary energies 
are included in the performance index alo'ng a path. This is a 9 stage 
pro'blem with up to^ six alternatives and up tO' three decisiO'ns per stage, 
This problem can be solved by considering it as a trajectory problein In 
which It is desired to move from A to B while minimizing the perfomaiice 
index to give the shortest path. At each junction designated by a circle, 
the alternatives are a motion upward to the right,, downward to the right 
or horizontally to the right. Starting at the final pO'lnt B^ as the 
.origin, there are four pO'SSdble preceding states,, and the value of the 
performance index is indicated next to the arrow. TO' go from stage 0' tO'' 
stage 1, we must find f for each decision and enter them in the circles. 
From stage 2 to stage 1, a decision is made for each node by minimizing 
f at each node and entering the value at the node. This step is shown 
In Figure 5. The complete analysis is sho^wn in. Figure 6 and the optimal 
trajectory path in Figure 7. The minimum value of the performance index 
for this path is 166. 

Utilizing the principle of optimality, one obtains a considerable 
reduction in the number of calculations necessary compared with coimplete 
enumeration O'f the possibilities. For a system with three choices at 
each junction as in the example, the number of calculations for dynamic 
progra,mmlng is: 

[N'^/3 + (N=2)]/2 
TO' enumerate all possible paths, 

(N-1) (N!) 



(N/3!) (N/3!) 

calculations are necessary, A 9 stage problem requires 125 calculations 
for dynamic progra,mming compared with 80,640' for enumerating all paths. 
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Figure 4. Process Alternatives 
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Figure 5. Initialization of Dynamic Programming Procedure 
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Figure 6. Evaluation of Trajectories 
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Figure 7. Optimal Trajectory Path 
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5. Minimi zat Ion of Pipeline Pumping Energy for Water TransmlssloTi, Systems 

An interesting application of dynamic prograinming is found in 
minimizing pumping energy o^f a pipeline system. Figure 8 is a simplified 
flow diagram of a series of pumping stations and the corresponding 
pressure distance profile. 

Assuiie that a flow Q is to be delivered from an inlet suction 
pressure (P ,)^ by a series of pumping stations L^^ miles apart to a teminal 
at pressure P . At the booster stations sufficient energy is added to Q 

to overcome friction losses in the pipeline as well as the elevation heads. 

, , th 
The pressure balance at the i station is J 

where AP_, = friction pressure drop/unit length 

between stations i and i-1 

L. - Li ■ - the length of line between station 1 and i-1 

E. - E. = the difference in elevation between 

stations i and i-1 

K = a prop'Ortlonality constant 
Knowing the initial pressure (?,)], and the terminal pressure P^, 
optimization is executed so that the sum of the energy requirements over 
N' stages is a minimum. This means finding the values of (Pg)^ and (P^) j. 
over the N stages. 
Define: 

Y. = the pressure boost across the i ' pumping station 



■^ Z 
^''- N pumping stations 



X = Z Y '= the cumulative pressure boost across the 
"'1 



f .(x) ^ the minimum' energy to deliver Q units/time through 
N 

the N pump stations when the sum of the cumulative 

pressure drop across the N stations is x, using an 
optimal policy. 

g^ (Y_ ) = the energy for pumping Q at station N. 



f ■ .'■■ 
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The functional equations for dynamic programming are: 

f_.(x) - min [g (Y ) + f .(x-Y )] (5,2) 

N n n n-1 n 

o ^ Y^ ^ X 

f^ = min [g^(Y^)] = g^(x) (5.3) 

o ^ Y^^ ^ X 

N 

At each stage of the process, there are constraints on the suction and 

discharge pressures : 

(P ). min ^ (P ), ^ (P ). max; i = 1, 2, ...,N (5. A) 
SI si SI 

(P^). min < (P.). ^ (P.). max; i = 1, 2, . . . ,N (5.5) 
d 1 d 1 d 1 

Execution of equations (5.2) and (5.3) must honour the constraints 

(5.4) and (5.5). 

The optimization may be carried out in many ways depending on 
the physical situation in the pipeline. Additional data are necessary 
such as the pump efficiencies, pipe diameters, friction factors, etc. 
Minimization may be affected by considering a variety of impeller sizes 
for each pump. In general, the procedure is similar to that of the 
example in section 4 except a greater number of decisions must be made 
at each stage. 
6 . Multi-Unit Water Treatment Plant 

A water treatment plant consists of a sec^uence of units made up 
of filters, precipitators, chemical mixers, reactors, adsorption bedsj 
etc. The effluent from one unit becomes the feed for succeeding unit. 
The entire complex may be considered as a multistage process in which 
the goal is to minimize energy, minimize several types of effluent 
contaminants or impurities, and minimize the quantity of chemical additives 
used. 
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Consider a general type of system where two feed streams 

F and F enter unit r, and effluent streams F F ,, and F 

A " B 1,E, N,t N-l,t 

leave 'units 1, N and N-1 reepectively as in Figure 9. 
Define the following terms: 

■y. . , = the flow rate leaving the I"'"' unit and going to the 
1 , 1— J. 

(i-l)th unit. 

(x, ) . . -. = the concentration of coinponent k in the stream 

^i,l-l' ^ ^ ^'■^■' " 

(S, ) . . -, = separation or removal factor for the kth component In 
k 1 , i+1 

the effluent stream F, .^^, mg/l of k/ total mgjl entering 

unit i- k ^ 1,2, ..... 

(S, ) . . = separation or removal factor for the kth component in 

effluent stream F, ._ , mg/£ of k/total rag/£ entering 

unit i ; k = 1,2, ..... 

(v ).. = the process variables, temperature, pres.sure,,, pH, etc., 
mi 

in the i '' unit. 

g. = the perforiB..ance of the i"' unit., involving contaminant 
reduction and operating energy. 
The separation or removal factors for the i ' unit may be evaluated as 

function of the entering streams, their compositions, and the pro'cess 
variables. 

(6.1) 

(6.2) 

Similarly, equations for mass baiances can be made around the rth unit 
with feeds F, and F^, around unit N-1 with product stream F and around 
units 1 and N, Component balances must be written for any chemical 
reactions using the separation factors. 
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Thos, the functiofial equation for ttie stage-O'iie unit Is: 

k 

^\^ k=l 
k = 1,2,... ; m = 1,2,. .. (6.3) 
For a twO' stage process the functional equation is: 

^2^^1,2" ^3,2' '^■''1^1,2,...' ^Vl„2' '^""1^3, 2,...' '^"^^3,2^ 



(6.4) 



.in [ Eg^CF^^^, F3^2' '^^2,3^ ^V2,l> + ^^ 



(v )« 1 1 
m / k=l 

k = 1,2,. ..; m = 1,2,...,. (6.5) 

In a similar manner the functional equations may 'be developed for the 

entire 1 stage process* 

The separation equations as well as the material balance equations 

are used in conjunction with the functional equations, to evaluate the 

'ininliium performance index (.see Appendix. A) . 

7. Discussion and^ Re.c.omne.nd.ations 

Conservation of energy has led tO' a rethinking of design 
proeedures for water and wastewater treatment systems.. This added dim.e.n.slon 
in design is another undetermined variable which must be minimised in 
addition to cost and effluent quality. The use of dynamic programming 
allows a sequential stage.wlse optimLzation of a very coimplicated system 
.In which energy should now be considered as an additional criterion In 
design. 

The main advantage to dynamic programniing lies in its se:quenti.al 
stage-by-stage character of computation. Of further import.ance is the 
fact that constraints on either the control or state variables actually 
simplify the comput.atlon rather than make it more difficult since the 
discretLzed range on the variables is reduced. 
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There are, of course, a number of problems associated with the 
use of dynamic programming. One of these has been called by Bellman the 
"menace of the expanding grid". This phenomenon is due to the generation 
of inadmissible states at the end of the state's discretization; as the 
calculation propagates, this may lead to many inadmissible values. The 
second problem, and the most serious one, results from the dimensions of 
the state variables. With a large number of state variables which are 
discretized at each stage of dynamic programming, a computer storage 
problem may result. This has been called by Bellman "the curse of 
dimensionality" . 

These problems are trivial in the light of the advantages and 
simplicity of dynamic programming. Careful formulation of the minimization 
procedure can eliminate them. 

As a tool in making optimal decisions for energy allocation 
and utilization, dynamic programming is recommended for future submissal 
of design proposals wherein several alternatives are considered as in 
Figure 3 and some sort of performance index is minimized with respect to 
operating costs, effluent specifications and energy requirements. 

Further reductions in energy requirements may be achieved by 
utilization of more recent developments in the unit operations design as 
well as innovative treatment systems. This will result in a wider range 
of alternatives which will aid the approach to energy conservation. 



, ^v» ..i=i?y.^. 



- 19 - 



EEFERllCES ; 

1. Athans, M. and Falb, P.L., "Optimal Co^ntrol",, McGraw-Hill Book Co., 

1966 . 

2. Bellman, R. , '"'Dynamic PrO'graiiming" Princeton University Press, 

1957.. 

3. Bellman, R, ,, '''Applied Dynamic Pro'gramming" ,, Princeton Untirerslty 

Press, 1962. ' .,.^ 

4. Lapidus, L.. , and Luus, R. , "Optimal Control of Engineering Processes" 

Blaisdell Publlsliliig Company, 1966. 

5. Zarnett, G.D., "Energy Requirements for Conventional and Advanced 

Wastewater Treatment", Publication No. ¥47, Applied Science-s " '• 
Section, Pollution Co^ntrol Branch,, Ministry of the Environment, 
Toronto', October 1975. 

6.. Zarnett, G.D.. , "Energy Requirements, for Wastewater Treatment 

Equipment", Teclinical lote No. TN 7008, Applied Sciences Section, 
Pollution Control Biranch, Ministry of the Environment, Toronto, 
July 1976. 

7. Zarnett, G.D., "Energy Requirements for Water Treatment Systems", 
Researcii Paper No, S2043, Applied Sciences Section, Pollution 
Control Brancli, Ministry of the Environment, Toroato, November 1976(a) 



20 - 



APPENDIX I 
EXAMPLE OF MATHEMATICAL OPTIMIZATION OF A TREATMENT SYSTEM BY DYNAMIC 

PROGRAMMING 

Consider a system with P discrete pollution treatment steps. 
If we consider a typical stage k, as in Figure Al, x(k) is the input 
concentration of wastewater contaminant and x(k+l) is the output concen- 
tration of stage k. Also, let u(k) be the energy input at stage k. 
AssiMne that we have sufficient knowledge of the system so that we have 
a system equation for any stage; 

x(k+l) - ax(k) + ku(k) 
with initial feed concentration known, as x(o) , and k - 0,1,2, ; and, 
a and b are constants. 

To optimize P stages of the system we must have a means to 

evaluate the performance. This is usually done using a performance 

index. We can formulate a performance index by saying that we wish to 

minimize the contaminant output and at the same time minimize the energy 

requirement. To penalize ourselves heavier, we should minimize the 

square of the concentration and the energy. If I Is a performance index, 

then : 

I ^ x^(k) + u"(k-l) 

Because we must minimize P stages, the performance index is of the form 

p 

I[x(o), P] - Z[x^(k) + u-(k-l)] 
k-1 

Thus, the problem is to find u(o) , u(l) , . . . . , u(P-l) such that I[x(o), P] 

is minimized. 

The solution using dynamic programming is as follows: The final 

state x(F) is taken to be the origin. First consider one stage, with P=l, 

and go from x(o) to x(l) by a selection of a(o) which minimizes I[x(o),l]. 
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Here: 



- 22 - 



I[x(o),l] = x^'(l) + u^(o) 



But, it is noted that the origin can be reached in one step, x(l)-0-oTigin. 

I[x(o),l] = u'^'(o) 

For equation, (4.1) to be satisfied , it is necessary that 

ii(o) = x(l) - ax(o) = a ^^^^ 
h b 

Note that this value of u(o) is a special case sinee it is independent 

of the performance index. Thus, there is only o^ne unique value of energy 

u(o), which connects x(o) to x(l) in one step. Knowing u(o) 

9 2 
I[x(o),l] = a X (o) 
b 

lextj co'nsider P=2 and find u(o) and u(l) which take the sys:tem from 

x(o) to x(2) while minimizing I[x(o),2]i. To keep the nomenclature 

consistent, x(2) = ^ origin, and: 

I[x(o),2] = x^'CD + u^(o) + x^'(2) + u^(l) 

Figure A2 shows a schematic of the steps for P=l and P^'2. Now 

x(2) - = ax(l) + bu(l) 

from which: 

u(l) = = (a/b) x(l) 

and thus : 

2 

I[x(o),2] - x^(l) + J(o) + u^(l) = (1 +-,)x^(l) + u^(o) 

b 
To obtain the extremum of I[x(o)j2] for a choice of u(o) , we can differ- 
entiate I|[x(o),21 and set the results equal to zero. 

2 
Il[x(o),2]|/u(o) = = 2u(o') + 2(1 +-^)x(l) ax(l)/au(o) 

Since ■ 

x(l) - ax(o) + bu(o) 3x(l)/a'u(o) = b 
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FIGURE A3. THREE STEPS IN DYNAMIC PROGRAMMING EXAMPLE 
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Hence: 

.2 
9l[x(o),2]/3u(o.) = = 2u(o) + 2(1 +- ) [ax(o) + bu(o)]b 

b 

or 

3 
. . ab + (a /b) x(o) 

u(o) = - ^ ^ 

1 + a" + b"- 
Thu,s x(l) „ u(l) and x(2) may be obtained. We can go on to a third stage 
tO' get x(o) to x(3) by a proper selection of uC'o), u(l) , and u(2) while 
I[x(o):j3] Is minimized. In this case x(3) =■ = origin,, and 

I[x(o),3] = x^(l) + u^(o) + x^(2) + u'^'(l) + u^(2) 
Figure A3 sho^ws the three stage pro'Cess. This process may be continued 
for P stages. 
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